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10 Background of the Invention 

This application continues from US application 10/350,386 filed 01/23/2003 
which continues in part from 08/482,944, filed 06/07/95, presently pending. These 
inventions are generally concerned with computer vision systems and specifically 
concerned with computer vision systems combined with computer aided design modeling 
15 facilities. 

The following disclosure is presented with the assumption that elements of 
computer aided design are well known to the reader. Texts explaining the current state of 
that are available from various sources, but those relating to AutoCad® may be 
particularly helpful. No attempt is made here to explain the details of implementing 

20 computer models with software except as it relates to the new arraignment presented. 

Vision systems which combine electronic cameras with computers to perform 
image functions are described in various texts including co-pending patent applications of 
the present inventors having series 08 serial numbers: 1 19,360; 270,91 1; 355,710; 
335,940; 335,912; 411,299; 307,360. 

25 It should be understood that all of the herein referenced provide considerable 

definition of elements of the present invention. Therefore, those materials are 
incorporated herein by reference whereby the instant specification can rely upon them for 
enablement of the particular teachings of each. 

Computer aided design and modeling techniques are useful in various fields. 

30 Commonly referred to as CAD or computer aided design, an electronic computer can be 
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used to model devices, objects or environments. Changes to the model are easily made 
and a great number of variations might be tried before arriving at a final desired result. 
Examples of uses of CAD include: by architects for buildings proposed to be built, or by 
automobile designers for cars proposed to be manufactured. A designer may start from a 
5 conceptual idea, a photograph, an artist drawing or other medium. From that initial idea 
or representation, a designer may construct on a computer, a model. A model consists of 
a plurality of individual graphical objects each having some correspondence to some real 
object. A photograph suggests to the CAD designer how particular features of the subject 
may look and give hints to how they might be implemented in a graphical representation. 
1 0 With advanced computers, it is even possible for the photograph to be scanned into an 
electronic image and combined with the CAD drawing, thereby facilitating formulation of 
a more accurate model. Converting real world objects into a computer model is of great 
interest to CAD designers as it provides very sophisticated designs in early stages of 
development. 

1 5 Novel techniques have been discovered which provide very specialized uses of 

vision systems, and in particular as they may relate to computer aided modeling and 
design. While the systems and inventions of the prior art are designed to achieve 
particular goals and objectives, some of those being no less than remarkable, these 
inventions have limitations which prevent their use in new ways now possible. These 

20 prior art inventions are not used and cannot be used to realize the advantages and 
objectives of the present invention. 

Summary of the Invention 

It is now first presented here, inventions of vision system computer modeling 
25 apparatus including devices for creating computer models of real scenes. It is a primary 
function of these systems to provide means and method for computer modeling or real 
objects. It is a contrast to prior methods and devices that known systems do not interact 
with real scenes with respect to perspective and spatial relationships as measured in real- 
time. A fundamental difference between the computer modeling apparatus of these 
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instant inventions and those of the art can be found when considering its interaction and 
spatial relationships with respect to scenes being modeled. 

An electronic imaging system combined with a computer aided design system and 
in communication with position, attitude, and optionally range measuring devices forms 
5 vision system computer modeling apparatus of these inventions. These vision system 
computer modeling apparatus are operable for addressing real scenes from some user 
chosen point-of-view 5 or perspective. The model presented at a display as a two- 
dimensional image has associated therewith a perspective which relates to a perspective 
of the three-dimensional real scene being modeled. The model as presented to a user is 
10 continuously responsive to changes in perspective. When the imaging system is moved 
such that the perspective of the scene changes, the perspective of the model similarly and 
correspondingly changes as well. This is accomplished via measurements of position and 
attitude of the apparatus with respect to the scene being addressed. 

Tools and principles of computer aided design can be combined with real time 
15 continuous imaging systems to result in sophisticated modeling instruments. As it is 
readily possible to measure spatial information including position, attitude and range of 
the imaging system with respect to scenes being modeled, these data can be used to better 
create advanced models with great accuracy with respect to the real world. 

For illustration by example, the combination of a computer running CAD 
20 software; a camcorder electronic camera; and position and attitude measuring apparatus 
forms the foundation of these systems. 

Using such a system, a model designer may view a scene in real time and 
construct a model of the scene as it is viewed. Computer graphical objects can be created 
and associated with attributes of the real scene. In particular, the graphic objects and their 
25 nodes can be indexed to the real position and orientation of objects and features of objects 
they represent. As the camera is moved about the scene, i.e. to a new perspective, so can 
the appearance and perspective of the graphical objects in the computer model in a 
fashion which corresponds to the way real objects would be imaged from the new camera 
position. By combining computer aided design with real time electronic cameras and 
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position, attitude and range measuring systems, a novel and powerful modeling system is 
realized. 

These inventions thus stand in contrast to methods and devices known previously. 
The invention is a computer modeling apparatus with live real-time images and position 
5 and attitude measurement devices which relate the computer model to the scene being 
modeled in proper perspective for all relative positions of the apparatus with respect to 
the scene. Presently CAD systems are typically computer workstations with the ability to 
scan photographs and other graphics which may relate to a scene being modeled. The 
position and orientation of the workstation has nothing to do with the models residing 
1 0 within the computer. 

It is a primary object of the invention to provide vision system computer modeling 
apparatus. 

It is additionally an object of the invention to provide computer aided design 
apparatus whose models are responsive to the position and attitude of the apparatus with 
1 5 respect to the scene being modeled. 

It is an object of the invention to provide apparatus comprising the combination of 
a computer aided design system, an electronic camera, position and attitude measuring 
apparatus and optionally, ranging apparatus. 

A better understanding can be had with reference to the detailed description of 
20 Preferred Embodiments and with reference to the appended drawings. These 

embodiments represent particular ways to realize the invention and are not inclusive of all 
ways possible. Therefore, there may exist embodiments that do not deviate from the 
spirit and scope of this disclosure as set forth by the claims, but do not appear here as 
specific examples. It will be appreciated that a great plurality of alternate versions are 
25 possible. 



Brief Description of the Drawings 

These and other features, aspects, and advantages of the present invention will 
become better understood with regard to the following description, appended claims and 
30 drawings where: 
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Figure 1 is line drawing of objects having a certain spatial relationship; 

Figure 2 is a different view of the same scene; 

Figure 3 shows a close-up and normal view at once; 

Figure 4 shows a resizing operation; 

Figure 5 shows a scene space geometry; 

Figure 6 shows a top view of that same geometry; 

Figures 7 and 8 illustrate line elements generated from various positions; 

Figure 9, 10, 1 1, and 12 show the side of a building being modeled; 

Figure 13 is a flow diagram for a modeling method; and 

Figure 14 is a system block diagram. 

Preferred Embodiments of the Invention 

In accordance with each of the preferred embodiments of these inventions, there 
are provided apparatus for creating a computer model of a scene where the model is 
responsive to the position and attitude of the apparatus. It will be appreciated that the 
apparatus of one preferred embodiment may be different than the apparatus of another 
embodiment. Many alternatives and versions may exist for various components of these 
systems. Adoption of certain variations can be made without deviation from the true 
spirit of the invention. The scope of the following disclosure should not be limited to the 
example elements chosen here for illustration. The scope of the invention is set forth by 
the appended claims. The following examples are considered by the inventors to be the 
best of many possible modes of executing these inventions. 

To realize an apparatus of these inventions, one may combine: 1) a camera; 2) a 
computer, and 3) position and attitude measuring devices. 

Real-time electronic digital video is available by way of common CCD type 
electronic cameras. Hand held camcorders can be battery powered and highly portable. 
They allow a user to view some real scene at a display device, for example a liquid crystal 
display device. In addition, they are particularly well suited for the present subject as the 
digital signal they produce is compatible with computer processing operations. 
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Computers use similar liquid crystal displays for user interface. The images 
computers generate therefore are compatible with presentation on those devices. 
Therefore the combination of electronic cameras and computers is facilitated by the 
common display format. Very advanced design software is now available having 
5 graphical modeling capabilities. CAD methodology permits graphical objects to be 
manipulated in a great plurality of ways so that they may represent real objects in a 
computer model. These present inventions therefore can most simply be described as the 
combination of a real-time electronic camera with a computer running CAD type 
modeling software. 

1 0 A user may address some real scene by pointing the camera there towards from a 

fixed position. Where a computer is connected to the camera and is running CAD 
software, a user may create a model of the real scene as it is viewed. 

A careful reader will realize that models created with such a system would 
necessarily be specific to a certain perspective. For this reason, the system computer is 

1 5 preferably arranged to be in communication with position and attitude determining 
devices. If the device is moved, then the perspective of the real scene necessarily 
changes. To account for this change in the model, the computer mathematically 
determines a new perspective for the objects of the model by applying rotation, scale, 
translation algorithms which are responsive to the camera's new position such that the 

20 perspective of the model continuously updates in accordance with the true perspective of 
the real scene. 

For example, a camera is addressing a scene, Fig 1, of a box (1 1) five meters in 
front of the camera and a ball (12) five meters further in the same direction. If the camera 
is moved toward the scene, Figure 2, the objects in the foreground, the box (21), would 

25 appear in the display to increase in size at a greater rate than those in the background, the 
ball (22). Given sufficient views of the objects in question, the apparatus would have 
sufficient data to recall a model of a cube (33) and a sphere (34) to match to the box (31) 
and the ball (32). The recalled models would then be scaled and oriented by the user or 
the apparatus itself, and placed so that the models (43, 44) coincided with the real objects 

30 (41 , 42) as is shown in Figure 4. 
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For the computer to properly size the two objects it may be required to re-size 
"stretch" the objects, either manually or automatically via image recognition for example, 
to fit the new size of the corresponding real box and ball. After such manual re-sizing, 
the computer could estimate via a triangulation routine the relative positions with respect 
to the measured camera displacement. Additional camera displacements could result in 
automatic graphic object updates. The first re-sizing operation would calibrate the device 
and further adjustments may be calculated and the re-sizing of the model objects 
performed without further user input. Similarly, other perspective and depth features 
could be accounted for. Mathematical translations from a real world 3-D scene to a 2-D 
computer display are already very common. Many advanced computer routines have 
been developed to facilitate perspective representations of 3-D objects on a 2-D display. 
It is important to realize here that the invention does not propose any new perspective 
computation routines. The well developed routines known are sufficient. It is, however, 
a new combination of primary elements which result in the powerful modeling tool. 

An alternate apparatus of the invention may comprise: 1) a camera; 2) a computer; 
3) position and attitude measuring devices, and 4) a range finding device. The addition of 
a ranging capability enhances the ability of the apparatus to create three dimensional 
models of the scene being addressed. To calculate the position of a point in the scene, the 
user would range on the point of interest and the unit, given position and attitude data, 
could calculate the coordinates of the point directly. It is important to realize that the 
invention does not propose a new method of position calculation based on unit position 
and attitude, and range to object. This method has been in use, in radar directed gunnery 
for example, for many years, though not for developing of computer graphical models. 
What is important is that range based position calculation enables the user to quickly 
develop a model based on nodes and connecting lines and curves, much as in CAD type 
programs. Also, range based position calculation enables the unit to use advanced 
modeling methods based on image processing such as edge detection within a defined 
area and coplanar modeling. 

For example, a user of the apparatus may choose to use a method of coplanar 
modeling, Figure 13. A user would range to sufficient points, at least three are required 
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to define a plane but the user could define as many points as are needed to define the area 
of interest, to define the boundaries of a planar area 1302. These boundaries may or may 
not be part of the model. The user may also define areas within these edge boundaries 
that the unit is to ignore. In other words, the user may cut pieces out of the defined area 
5 that will not be analyzed by the unit, in effect setting up interior boundaries. This would 
enable the user to define exactly the area that is to be analyzed by the unit. Once the area 
has been defined the unit then calculated the equation defining the plane 1303. The unit 
then detects, using edge detection and other methods known, all lines and curves within 
the defined area 1304 and displays them to the user 1305. The user then deletes all lines 

1 0 that are not to be modeled 1 306. By utilizing the knowledge of the location and 

orientation of the defined plane in relation to the units position, and the ability of the unit 
to measure angles from its bore-sight, by pixel counting and knowledge of lens zoom 
state, the unit can calculate the location of all of the nodes needed to define those lines 
and curves in the defined plane 1308, 1309. Figures 9-12 illustrate basic steps involved 

1 5 in relation to modeling a wall of an office building. In Figure 9, the user defines a planar 
area by ranging to vertices 1-6. Reference with respect to the users position (which is 
known to the computer) sets the precise location of the addressed point in 3 space. In 
Figure 10, a user confirms the process by using an interactive graphical user interface. In 
Figure 12 the unit displays the calculated model to the user. 

20 In figure 14, major components of these systems are illustrated in a block 

diagram. A camera, or cameras 141 are in electrical communication with a computer 
145. Measurement systems for position 142, attitude 143 and range are arranged together 
as part of a sensor subsystem. A video display 146 is also electronically coupled to the 
computer are serves to display images generated therein. Data store 148 is also a part of 

25 the computer subsystem and contains information such as computer models both 
generated and those stored from previous sessions. 

A continuation of the coplanar method would be to use a pair of planar models, 
created from different positions, using the same boundary limits. If any objects projected 
from, or were inset into, the plane the two images would appear different. The unit 

30 would be able to calculate, through triangulation, the correct coordinate for these nodes. 
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Some situations may require a third view from another location for completeness. Figure 

5 shows a general situation in which a user defines a planar area 51 that has a line 52 
sticking out of it at right angles. The viewer would go through the planar modeling 
process from viewing position #1 53 and from viewing position #2 54 to generate the 

5 coordinates for node A 55 within the plane and to calculate the coordinates of the 

endpoint, node B 56, without the user having to range to either point specifically. In this 
simple situation it would seem easier to just range on the two points in question, nodes A 

6 B, and have done with it. But if the area contained many projections or indentations 
the user, by using this method, would merely have to define a planar area, and view it 

10 from several points of view, to model these projections in three dimensions. Figure 6 

gives a plan view of the situation and clearly shows that, if node B 62 is assumed to be in 
the defined planar area 61, that the coordinates calculated for the intersection of a line, 
from the two viewing positions 63,64 to node B 62, and the plane 61 will produce 
different coordinates, nodes Bl 66 and B2 65. In the case of node A 61 the change in 

1 5 viewing position will not produce a change in coordinates because node A 6 1 is indeed in 
the defined plane. These coordinates, as calculated from each viewing position, are 
shown in Figures 7 and 8. These figures clearly shows that node A 71, 81 is within the 
plane. Figures 7 and 8 also show, through the discrepancies in the positions of modes Bl 
72 and B2 82, that the real position of node B is not within the defined plane. To 

20 calculate the actual coordinates of node B is quite simple. Given the knowledge of the 
location of each viewing position and the knowledge of the bearings, both horizontal and 
vertical, to the node, based on the pixel counting angular offset method, from each 
viewing position the unit could calculate the actual position of the node by triangulation. 
The primary elements of the systems are: 

25 a camera to acquire a digital image of a real scene; 

a computer to run CAD type software and combine graphical objects with the real 
image, and further to compute perspective relationships between a real 3-D scene and a 2- 
D perspective of that scene, the computing routines being responsive to position, attitude 
and ranging measurements; and 

30 apparatus operable for measuring position, attitude, range. 
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More specifically: 

the camera may be a camcorder type, a portable battery operated device for 
producing a digital video signal, in communication with a computer processor; 

the computer may be the type known as a personal computer: having a 
5 microprocessor for example Pentium or Power PC processors which have been employed 
to execute code developed for computer modeling, further being operable for combining 
an image from an electronic camcorder with a computer generated graphical model 
image, further being in communication with and responsive to apparatus for measuring 
position, attitude and range of the camera. 
10 The apparatus for measuring position, attitude and range may be as follows: 

the position measuring apparatus may be a large scale system such as the global 
position system G.P.S. or may be a local system arranged with a particular environment in 
mind such as a city block or single room of a building, each employing some radio 
location measures, depending upon desired accuracy, the choice of appropriate position 
1 5 measuring system will vary; 

the attitude measuring apparatus may be an electronic compass, flux gate 
compass, interferometer ring gyro or other device which is operable for measuring 
relative pointing direction of the camera imaging axis; and 

the ranging apparatus may be an ultrasonic or radar device which measures 
20 relative position of object by timing a test pulse which is reflected therefrom. Some of 
these devices are common to simple photographic cameras in auto-focus systems. Other 
types of ranging, such as laser or infrared rangefinders, may be equally effective as long 
as it is possible to determine the distance from the camera to a particular object and relay 
that measurement to the computer. 
25 In one special version of these inventions, cameras are omitted. A viewing path is 

defined by a reticule system which provides reference or "pointing" axis for the devices. 
In common CAD methods, "pointing" and "clicking" may be done via a mouse peripheral 
device. An icon, for example an arrow, designates a position on the model field as 
displayed on the screen. Objects of the model are manipulated by pointing and clicking, 
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clicking and dropping, etc. The model may relate to some real scene, but is not generally 
responsive thereto. 

In a particular version of the invention, pointing, clicking and dragging operations 
may be implemented by manipulation of a pointing axis into a real scene. Clicking is 
5 accomplished by a simple switch on the device sensitive to tactile manipulation. Again, 
the system's ability to know its position and pointing attitude as well as certain range data 
allows it to directly interface with the real world. The model is sensitive to and 
responsive to the position and attitude of device in the 3-D scene. The displayed model, 
therefore, corresponds to how the scene would look from any perspective. 

10 Modeling methods of these inventions include methods having the primary steps 

as follows: viewing a scene with an electronic camera; measuring position and attitude of 
said camera; recording a first point associated with said measurements; changing either 
the position state or attitude state of the camera; recording at least one other point 
associated with the new position and attitude state; and displaying said points 

15 superimposed with an image captured with said electronic camera. In this way, one can 
see the scene while also viewing the model being formed together in the same 
perspective. The methods may also include a step where the points are connected 
together by line segments. Or, further by connecting several points to form a polygon 
computer model object. Such polygons may be shaded, colored, or textured by 

20 application of these attributes in the computer model databases. 

Where a model of an object in a scene is formed from one point of view, the user 
may later go to a different point of view and repeat some or all of the points. From a 
second position, the points which are repeated may be averaged with the points acquired 
from the previous viewpoint to form a more accurate point definition. 

25 Although the present invention has been described in considerable detail with 

clear and concise language and with reference to certain preferred versions thereof 
including the best modes anticipated by the inventor, other versions are possible. 
Therefore, the spirit and scope of the appended claims should not be limited by the 
description of the preferred versions contained therein. 
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